Evaluating the quality of an assessment: is the OSCE a valid and reliable assessment of clinical skill? [abstract]

In order to ensure horizontal integration, the curriculum at the Faculty of Medical Sciences, St. Augustine, includes a formalized skill lab programme in which student are taught basic clinical skills. The programme is measured at the end of year by a summative OSCE. Over the years, questions have been raised about the quality of the skills lab programme as well as the examination and its place in the curriculum. On the other hand, there is a movement towards greater use of OSCE in Phase 2. This study reports the use of a unique evaluative framework to assess the quality of the OSCE as an assessment tool. Key indicators are derived from the assessment literature and questions from a number of instrument along with new items are collated in an evaluation instrument. Key indicators include (1) construct validity, (ii) content coverage, (iii) impact upon students, (iv) fairness, (v) authenticity, (vi) reliability, (vii) usability and organization, and (viii) transparency. Evaluation data for the Year 3 OSCE (1998-2000) and OSCE for Medicine and Therapeutics are included. In line with the trend towards considering student perceptions and satisfaction with the teaching, learning and assessment provision, the focus is on the evaluation of student views, using both open and close-ended questions. However, performance scores are also considered when assessing the year 3 OSCE. Data suggest that the OSCE at the end of Phase 2 has a positive impact upon facilitating learning in Phase 2. Indeed, there is a moderate to strong relationship between performance in the OSCE and subsequent performance in Phase 2. In comparing data from recently completed OSCE in Phase 2 Medicine and Therapeutics, the possible negative impact of the high stakes nature of the examination is noted. The study recommends greater use and better structuring of this examination mode. (AU)

Effectively assessing learning in the Basic Health Sciences: using different modes of assessments-what do they measure? [abstract]

In an integrated problem-based learning focussed curriculum where there are significant levels of vertical and horizontal integration, for assessment to be appropriate and aligned a variety of assessment modes must be utilized. At the Faculty of Medical Sciences, significant curriculum changes have been accompanied by changes in the assessment scheme. One of the major significant changes has been the use of a varied but appropriate assessment mode when constructing continuous assessment. Increasingly, Faculty has made use of both construceted response and performance assessments in addition to the traditional selected response formats. In terms of quality assessment schemes, such policies reflect an improved assessment scheme that is likely to more effectively fit both the subject area and the changing student profile. This study considers the predictive validity of different assessment modes using the correlation with the student's admission scores. The study includes an investigation of the validity of: Constructed response formats such as the recently introduced Modified Essay Questions (SAQ's), both used in Continuous Assessments as well as the Restricted Response Essays used in the Phase Examinations of the New Programme. Selected response formats such as the MCQ and MTF. Performance Assessments, including scores derived from laboratory reports and seminars. It was found that selected response formats were generally of higher validity, suggesting perhaps greater rigour and fairness, despite the possibility of test-wiseness and cueing as factors limiting construct validity. Concerns over standardization and rigour in both performance and constructed response assessments are identified. Recommendations are provided for developing greater rigour when implementing performance assessments and for establishing feedback systems based on scores from constructed repsonse formats. (AU)

Developing and using performance indicators in evaluating quality and institutional effectiveness: the Faculty of Medical Sciences, UWI, St. Augustine [abstract]

Performance indicators are quantitative indices that describe organizational performance. They are frequently used for assessing institutional effectiveness and in performance management. In the current climate of accountability in higher education, performance indicators have an increasingly significant role in evaluation and benchmarking. In the United Kingdom (UK), the quality assurance agency has recently produced a range of performance indicators for all UK universities. Likewise, at the Office of Planning, the University of the West Indies, in its mid-term review makes use of performance indicators in assessing student progression in the key faculties at St. Augustine and Mona. However, traditional performance indicators may have limited interpretive value as evaluation indicators and benchmarks when applied to the assessment of institutional effectiveness of the Faculty of Medical Sciences. This lack of validity might relate to the unique structure of this Faculty, including the diversity in the student population, high admission scores, balanced gender ratios, a five-year multi-phased programme, a variety of assessment modes, and the facility for resits which aid academic progression. This study reports on an attempt to develop a compendium of performance indicators for use in quality management at the Faculty of Medical Sciences, St. Augustine. Data from all Phases and for all graduating classess in the medical and veterinary schools are used to evaluate the validity of a variety of access and progression indices, including value-added data and failure rates for the class of 1994 to 2000. The design of valid post-qualification indicators and a further study of non-completion are consisdered and rationalized. The study confirms limitations in the validity and reliability of common performance indicators and a lack of rigour and benchmarking status when applied to the complex programmes in the Faculty of Medical Sciences. It is argued that (1) greater attention must be payed to access indicators related to diversity, (2) progression indices should be varied and contextualized, and (3) performance indicators should include value-added indices which will allow progression rates to be compared with entry characteristics, including gender. (AU)

The assessment of empathy and the quality of interpersonal communication by standardized patients in an OSCE: an exploratory study [abstract]

Emphatic communication is a critical aspect of medical interviewing. While there are a number of theoretical perspectives on empathy, in the medical context Feignhny (1995) describes, empathy as, "a physician's cognitive capacity to understand a patient's needs, an affective sensitivity to a patient's feelings, and a behavioural ability to convey empathy to a patient". From this definition, the most direct and valid assessment of empathetic concern is likely to come from the patients own perspective. The development of empathy in students is critical in the provision of quality undergraduate medical attention. At the St. Augustine Campus of the Faculty of Medical Sciences, the skills laboratory programme has the responsibility for developing basic clinical competencies in medical students in years 1 to 3, including empathetic concern. However, as yet, there are no formal intervention programmes. A standardized patient (SP) programme is maintained in which a core of volunteers is trained to work with the students and to perform in the role of patient at the annual OSCE at the end of year 3. In the 1999 OSCE, the SPs were asked to assess the quality of the interpersonal communication of students. These ratings were compared with those of experts raters at key stations. In the 2000 OSCE, the study was expanded to include an assessment of empathy using the Adjective Check List. The key research questions for the study focused on the relationship between empathy and student individual characteristics and performance in traditional assessments. It was found that empathetic communication as practised in the OSCE and as assessed by the SPs was related to gender but unrelated to performance in traditional assessments in the Phase examinations and admission scores. There was evidence that students with high admission scores were likely to score lower in some aspects of empathy as perceived by the SPs. Recommendations for the development and assessment of attitudes are provided. Issues centring on design intervention programmes for developing empathetic concern in students are discussed. (AU)